Ok, so you want to evaluate and open source application?

What guidelines should you use? Here is a guideline. I will continue to update this as I find valid items to measure. If you have something I should add to the list, please let me know.

Obtaining the Software

1. A top link in search engine when searching for open source app’s name?
2. A quick download link?
3. Clear description of different downloads per platform?

Installation of Open Source App

1. Clear description of different downloads per platform?
   List of platforms:

2. Ease of install score:
3. Ease of initial configuration score:

Authentication

1. Integration with Active Directory?
   Score:

2. Integration with LDAP?
   Score:

3. Database authentication?
   Explanation: Can authentication occur in a database such as Postgresql, MySQL, etc…
   Supported Database list:
   Score:

4. Authentication to a 3rd party programs database?
   Explanation: So that if you have an application A with a database that hosts a username a password, can this open source application B use your database from application A to authenticate?
   Score:

Security

1. How secure is this application?
2. What security holes have been reported and fixed?
3. What development designs were taken into consideration to enhance security?
4. What security analysis tools such as Nessus has this open source application been analyzed with?

Documentation

1. Install guide exists?
   Quality Score:

2. Users guide exists?
   Quality Score:

3. Admin guide exists?
   Quality Score:

4. Developer’s guide exists?
5. Compile/Debug guide on how to load in an IDE and compile and debug (Visual Studio 2008, Eclipse, KDevelop, other, etc…)
6. Guide for submitting a bug or suggestion?
7. Guide for contributing documentation?
8. Ease of contribution Documentation?

Ease of Use

1. Is the application easy to use?
2. Can non-technical users use the application with minimal training?

Stability

1. How stable is the application? Determine this from normal use for a period of time.
2. How stable is the platform(s) and/or 3rd party dependencies the application runs/depends on?
3. Does the application crash with normal use?
4. Does the application crash with abnormal use?
5. Does the application crash with prolonged use?
6. Is the process for submitting a bug simple?
7. Is the process for applying a patch simple?
8. Does applying patches decrease stability?

Community Strength

1. Is it being maintained by a strong community?
2. Is there a high adoption rate for this application?
3. What is the average turn around time for a bug in the community?
4. Is there a forum? What is forums user base? How quick do questions get responses?
5. Is there a mailing list?
6. Is there an RSS feed?

Customization of Open Source Application

1. What language is this written in?
2. Ease of customization.
3. Ease of contributing to project
4. Ease of compiling/debugging?
5. Ease of getting fixes committed to source?

Scalability

1. Does the application scale well with increased usage?
2. Does this application integrate with the two most used operating systems for desktops? Windows and OS X?